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Executive Summary 


The NASA Protoflight Research Initiative is an internal NASA study conducted within the Office 
of the Chief Engineer to better understand the use of Protoflight within NASA. Extensive literature 
reviews and interviews with key NASA members with experience in both robotic and human 
spaceflight missions has resulted in three main conclusions and two observations. The first 
conclusion is that NASA’s Protoflight method is not considered to be “prescriptive.” The current 
policies and guidance allows each Program/Project to tailor the Protoflight approach to better meet 
their needs, goals and objectives. Second, Risk Management plays a key role in implementation 
of the Protoflight approach. Any deviations from full qualification will be based on the level of 
acceptable risk with guidance found in NPR 8705.4. Finally, over the past decade (2004 - 2014) 
only 6% of NASA’s Protoflight missions and 6 % of NASA’s Full qualification missions 
experienced a publicly disclosed mission failure. In other words, the data indicates that the 
Protoflight approach, in and of it itself, does not increase the mission risk of in-flight failure. The 
first observation is that it would be beneficial to document the decision making process on the 
implementation and use of Protoflight. The second observation is that If a Project/Program chooses 
to use the Protoflight approach with relevant heritage, it is extremely important that the 
Program/Project Manager ensures that the current project’s requirements falls within the heritage 
design, component, instrument and/or subsystem’s requirements for both the planned and 
operational use, and that the documentation of the relevant heritage is comprehensive, sufficient 
and the decision well documented. To further benefit/inform this study, a recommendation to 
perform a deep dive into 30 missions with accessible data on their testing/verification methodology 
and decision process to research the differences between Protoflight and Full Qualification 
missions’ Design Requirements and Verification & Validation (V&V) (without any impact or 
special request directly to the project). 



Introduction 


The NASA Protoflight Research Initiative is an internal NASA study conducted within the Office 
of the Chief Engineer to better understand the use of Proto flight within NASA. NASA has been 
using the term Protoflight since Goddard Space Flight Center (GSFC) coined the term in the 1960’s 
to “avoid overrunning budget limitations” along with other factors. The first designated Pro to flight 
spacecraft was launched in November 1965 and was called the Direct Measurements Explorer 
(DME-A). According to the first General Environmental Test Specification for Spacecraft and 
Components (GSFC S-320-G-1) published in 1969, a Proto flight spacecraft was defined as “(1) 
either a spacecraft originally designated as a prototype and subjected to complete or partial design 
qualification environmental testing, and then designated for flight use or (2) a spacecraft 
designated in advance to serve both as a prototype and a flight model.” [1] Today’s definition, 
located in the General Environmental Verification Standard (GEVS) for GSFC Flight Programs 
and Projects (GSFC-STD-7000A), defines Protoflight as “flight hardware of a new design; it is 
subject to a qualification test program that combines elements of prototype and flight acceptance 
verification; that is, the application of design qualification test levels and flight acceptance test 
durations.” 

While the definitions of Protoflight have not varied much in the past 50 years, the use of Protoflight 
has significantly increased. From 1965-1974, a total of 16 Proto flight spacecraft were launched 
compared to a total of 49 Protoflight missions launched from 2004-2014. The increased use of the 
Protoflight approach for larger more complex spacecraft, subsystems and instruments causes one 
to ask the question, does using the Proto flight approach increase the likelihood of mission failures? 

To answer this question and others, the authors of this paper conducted an extensive literature 
review encompassing the current policy and standards used within NASA, the number and type of 
NASA missions flown in the past 20 years and the number of missions that were exclusively 
Proto flight or that used one or more Proto flight subsystems/instruments in the past 1 0 years. Given 
the vast amounts of information regarding this topic and the various deep dives that could take 
place, the authors restricted the research to three areas each with a stated purpose to properly scope 
the literature review. The three areas and their purpose are as follows: 

1. Conduct a literary review of the current Policy and Standards at the Agency and Center 
levels for Protoflight programs and projects within NASA. Other US Agencies, foreign 
space programs, and companies’ policies/procedures/standards regarding Protoflight are 
not actively sought in this review. The purpose of the literary review is to understand what 
policies/standards are in place at the Agency and Center levels with a focus on when the 
Protoflight model began being used and the current guidance on the use of Protoflight 
models versus a full qualification model. 

2. Collect and analyze data on NASA’s flight missions and projects that have flown in the 
past 10 years using the Proto flight methodology at the payload, subsystem and/or 
instrument level. The purpose of this data retrieval and analysis is to understand the 
program/project’s decision to use the Protoflight approach and the rate of failure (partial 



and/or full) tracing back to a reduced life time expectancy of the 
payload/subsystem/instrument due to the use of the Protoflight methodology. 

3 . Characterize the level of Proto flight verification and Risk Management techniques used by 
NASA’s Protoflight missions/projects that have flown in the past 20 years. The purpose of 
this characterization is to determine the various implementations of the Protoflight 
approach used in NASA’s Protoflight missions/projects and the effect of the Protoflight 
approach used on Risk Management (technical and programmatic). 


Current Protoflight Guidance 


An extensive literature review was conducted to locate and obtain NASA’s Agency and Center 
level documentation regarding Protoflight qualification standards, requirements, policies, and 
guidance. Public and non-public sources were used to locate documentation. Non-NASA 
documentation was not actively sought, however if non-NASA documentation was located and 
obtained it was included in this study. Table 1 shows the various documents obtained including 
the document number, title, published and expiration date (where applicable) and the responsible 
center/organization. The documents obtained cover a broad range of infonnation regarding the 
Protoflight method including design levels, qualification levels for Protoflight, 
Prototype/Qualification and Acceptance, NASA policy regarding Risk Classification and the use 
of the Protoflight approach verification test requirements/matrix, see Table 2. 



Table 1. A list of the obtained NASA’s Agency and Center level Protoflight and/or ProtoQual Qualification 
Documentation including a Non-NASA sourced Department of Defense (DoD) Handbook including all applicable 
dates and responsible Center. 


Document No. 

Title 

Published 

Expires 

Center 

GSFC-STD-7000 
REV A 

General Environmental Verification 
Standard ( GEVS) for Flight Programs 
and Projects 

4/22/2013 


GSFC 

APR-8070.2 

Class D Spacecraft Design and 
Environmental Test 

9/17/2008 

9/17/2013 

ARC 

MSFC-HDBK-670 

General Environmental Test 
Guidelines (GETG) for Protoflight 
instruments and experiments 

6/1/1991 


MSFC 

NASA-STD-7002A 

Payload Test Requirements 

9/10/2004 


HQ 

NASA-STD-7003A 

Pyroshock Test Criteria 

12/20/2011 


HQ 

NASA-STD-7001A 

Payload Vibroacoustic Test Criteria 

1/20/2011 


HQ 

NASA-STD-5001B 

Structural Design and Test Factors of 
Safety for Spaceflight Hardware 

8/6/2014 


HQ 

NPR 8705.4 

Risk Classification for NASA 
Payloads (Updated w/change 3) 

6/14/2004 

6/14/2018 

HQ 

DOD-HDBK-343 

(USAF) 

Design, Construction and testing 
requirements for one of a kind Space 
Equipment 

2/1/1986 


DOD 

JSC-65828 REV A 

Structural Design Requirements and 
Factors for Spaceflight Hardware for 
Human Spaceflight 

10/1/2011 


JSC 

SSP41172 REV U 

Qualification and Acceptance 
Environmental Test Requirements ISS 
Program 

3/28/2003 


JSC 

JPL Rules! DocID 
55833 Rev 1 

Spacecraft System Dynamic and Static 
Testing, Rev. 1 

4/15/2013 


JPL 

JPL 

System Thermal Testing, Rev. 2 

5/8/2013 


JPL 




Table 2. A list of the obtained NASA’s Agency and Center level Protoflight Qualification Documentation including a Non-NASA sourced Department of Defense 
(DoD) Handbook including the subject matter covered in the document. (Yes means the topic was covered in the document. No means the topic was not covered 
in the document) 


Document 

No. 

title 

Protoflight 

approach 

(policy) 

Desig 

n 

Levels 

Verification 

Test 

Requirements 
/ Matrix 

Prototy 
pe/Qua 
1 Levels 

Protofli 

ght 

Levels 

Acceptan 
ce Levels 

Structural 
Loads (level & 
duration) 

Acoustics 
(level & 
duration) 

Random Vibe 
(level & 
duration) 

Sine Vibe 
(level & 
duration) 

Mech 
Shock 
(actual & 
sim) 

Therm- 

Vac 

(Max/Mi 

n) 

Thermal 

Cycling 

(Max/Min) 

EMC & 
Magneti 

cs 

Contaminat 
ion Control 

End to 
End 

Testing 

GSFC-STD- 
7000 REV A 

General Environmental 
Verification Standard 
(GEVS) for Flight Programs 
and Projects 

None/ 
Goddard Std 
Approach 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

APR-8070.2 

Class D Spacecraft Design 
and Environmental Test 


Yes 

Yes 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

MSFC-HDBK- 

670 

General Environmental 
Test Guidelines (GETG) for 
Protoflight Instruments 
and experiments 


No 

Yes 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

NASA-STD- 

7002A 

Payload Test 
Requirements 


No 

Yes 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

NASA-STD- 

7003A 

PyroshockTest Criteria 


No 

No 

Yes 

Yes 

Yes 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

NASA-STD- 
700 1A 

Payload VlbroacousticTest 
Criteria 


No 

No 

No 

Yes 

Yes 

No 

Yes 

Yes 

No 

No 

No 

No 

No 

No 

No 

NASA-STD- 
500 IB 

Structural Design and Test 
Factors of Safety for 
Spaceflight Hardware 


Yes 

No 

Yes 

Yes 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

NPR 8705.4 

Risk Classification for 
NASA Payloads (Updated 
w/change 3) 

Policy 

governing the 
use of the 
Protoflight 
approach 
based on Risk 
Class 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

DOD-HDBK- 
343 (USAF) 

Design, Construction and 
testing requirements for 
one of a kind Space 
Equipment 


No 

Yes 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

JSC-65828 
REV A 

Structural Design 
Requirements and Factors 
for Spaceflight Hardware 
for Human Spaceflight 


Yes 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

No 

SSP 41172 
REV U 

Qualification and 
Acceptance Environmental 
Test Requirements ISS 
Program 


No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

JPL Rules! 
DodD 55833 
Rev 1 

Spacecraft System 
Dynamic and Static 
Testing, Rev. 1 

None / JPL Std 
Approach 

No 

No 

No 

Yes 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

No 

No 

No 

No 


System Thermal Testing, 
Rev. 2 

None 

Yes 

No 

No 

No 

No 

No 

No 

No 

No 

No 

Yes 

Yes 

No 

No 

No 


The majority of the documents obtained give guidance in the form of technical standards, 
requirements, design criteria and/or test criteria without programmatic guidance. Some of the 
documents are NASA standards that pertain to specific technical areas such as pyro shocks and 
vibro acoustics. Others are aligned with the General Environmental Verification Standard (GEVS) 
for Flight Programs and Projects, GSFC-STD-7000, which covers a broad range of technical 
information. There is one key difference between the GEVS and other Center’s Proto flight 
documentation based on the GEVS. The GEVS states “The standard is written in accordance with 
the current GSFC practice of using a single Protoflight payload for both qualification testing and 
space flight... The Protoflight verification program, therefore, is given as the nominal test 
program.” In other words, Full Qualification is the exception and no longer the nonn at Goddard 
Space Flight Center (GSFC). 

Only one document attempts to give guidance regarding the use of Protoflight through 
“recommended requirements” based on risk classification, NPR 8705.4 Risk Classification for 
NASA Payloads. NPR 8705.4 Appendix C - Recommended SMA-Related Program Requirements 
for NASA Class A-D Payloads, states the level of rigor recommended to qualify flight payloads 
characterized with a Class A, B, C or D risk classification. Currently GSFC and JPL use the 
Protoflight verification program as the nominal approach for non-heritage flight payloads with a 
risk classification of Class B - D, which agrees with the “recommended” requirements in NPR 
8705.4 Appendix C. For missions with a risk classification of Class A NPR 8705.4 states “Full 
formal qualification and acceptance test programs and integrated end-to-end testing at all hardware 
and software levels.” However, the requirement is only “recommended” and does not require a 
waiver if a program/project does not comply. For an example, Mars Science Laboratory (MSL) a 
Category 1, Risk Class A Robotic Mission used the Protoflight approach on specific 
subsystems/instruments but not the entire payload. While a waiver was not required, the decision 
to use the Protoflight approach was acknowledged in MSL’s Key Decision Point (KDP) 
documentation. 

The Protoflight method within NASA is not considered to be “prescriptive.” The current policies 
and guidance allows each Program/Project to tailor the Protoflight approach to better meet their 
needs, goals and objectives. For example, some projects/missions use Engineering Test Units 
(ETUs) to test various subsystems/instruments prior to building the Protoflight Unit while others 
do not. Another example of variations between Protoflight projects is the decision to test or to use 
analysis to verily requirements during the qualification process. The decision process regarding 
the various implementation options of the Protoflight approach is based on several factors such as 
the heritage of subsystems/instruments, budget, schedule and risk posture of the project. The 
decision process is not required to be documented only agreed to by the decision making 
authorities. 



Protoflight Mission Failures in the Past 10 and 20 years 


Over the past 20 years, NASA has launched 215 missions varying from Category 3 Risk Class D 
missions all of the way to Category 1 Class A missions to Mars and launching humans into Lower 
Earth Orbit (LEO). Information regarding NASA’s missions over the last 20 years was gathered 
and reviewed for any links to the use of the Protoflight methodology at the spacecraft, payload, 
subsystem, and/or instrument level. In some cases, infonnation was not readily available for 
review. If enough information was not accessible the mission was noted as such. All accessible 
mission related documentation was reviewed for the following information: 

• Mission Risk Classification 

• Qualification Method used (Protoflight, Full Qualification) 

• Launch Date 

• Current Phase (Operating, Operating/Extended, Past, Past/Extended 

• Government Agency and NASA Center involved 

• Why the Protoflight approach was chosen 

o Information regarding this question was not available 

With the following definitions for clarification: 

Operating Phase : A period of time in which the operational activities are carried out to 
meet the goals of the mission (Operational Phase) 

Operating/Extended Phase : A period of time after the planned operational activities have 
been carried out and additional operational activities are being carried out (Extended 
Operational Phase) 

Past Phase : The period of time after the mission has been decommissioned according to 
plan or because of a significant failure and without the mission entering into an Extended 
Operational Phase 

Past/Extended Phase : The period of time after the mission has been decommissioned the 
mission entering into an Extended Operational Phase 


The authors categorized a mission’s qualification method as Protoflight only if all of the below 
Protoflight criteria were met. 


Protoflight Criteria 


1. Mission spacecraft, subsystems and/or instruments were launched and operated in space 



For STS missions, the only missions included are those that are external and fixed 
to the ISS or that are released into space 


2. A known Protoflight qualified spacecraft hub, payload, subsystem and/or instrument is on 
board 


- If specific spacecraft, subsystem, or instrument information is unavailable but Risk 
Mission Class is known to be B, C, or D, the mission is automatically assumed to 
be Proto flight based on NPR 8705.4 Appendix C 

• Class A missions are automatically assumed to have gone through Full 
Qualification based on NPR 8705.4 Appendix C 

3. The above criteria must have valid references, which include: 

- NASA personnel who worked the mission communicating via a face to face 
meeting, telephone call or email 

- NASA published (internal or external) documentation (NASA website, NASA 
Case Study, NASA Lessons Learned and/or NASA personnel author of a 
scientific/engineering journal article) 


Once a complete list of all of NASA’s missions over the past 20 years was collected and 
documented, the launch date, current phase, risk classification, qualification method and 
Government Agency/Center involved was determined through literature reviews of public and 
private accessible documentation. Documentation regarding the risk classification and 
qualification method used was not readily accessible for missions launched between 1994 and 
2004, resulting in a data set too small for high fidelity analysis. While a large amount of 
information regarding NASA missions from 1994 - 2004 was inaccessible, the data that was 
collected is included for the reader’s edification. Information regarding launched NASA missions 
during the last 10 years, 2004 through 2014, was readily available and allowed for a large enough 
set of information for analysis. For an example, Table 3 shows the total number of missions 
launched from 1994 - 2014 as 215 missions with only 110 missions having enough accessible 
information to determine the qualification method used. In the last decade alone, 2004-2014, 83 
out of the 84 NASA missions launched had enough accessible information to determine the 
qualification method used. Table 4 is an example of the collected mission information. 

Information on the launched Proto flight missions within the last 20 years (1994-2014) was 
collected and reviewed for mission failures with and without direct li nk s to the Protoflight 
qualification method. In this paper and in accordance with NPR 862 1.1B NASA Procedural 
Requirements for Mishap and Close Call Reporting, Investigating, and Recordkeeping, a “NASA 
mishap is an unplanned event that results in NASA Mission Failure before the scheduled 
completion of the planned primary mission. Where a NASA Mission Failure is a mishap of 
whatever intrinsic severity that prevents the achievement of the mission's minimum success criteria 
or minimum mission objectives as described in the mission operations report or equivalent 



document. (Note: A mission failure applies only to a NASA program's mission, and not a test or 
ongoing institutional operation. If a program accomplishes all minimum success criteria but not 
"full mission objectives," it is not a mission failure (even though in some cases it may appropriately 
be classified and investigated as a close call).” If a mission failure occurred a Failure Review Board 
(FRB) and/or Mishap Investigation Board (MIB) was created and a report released describing the 
failure/mishap and known and/or suspected cause. These reports were used to determine if a 
mission failure/mishap occurred on a Protoflight mission and the origin of said failure/mishap. If 
a mishap was found to occur on a Protoflight mission the data was collected and the report 
reviewed to determine any links to the Pro to flight qualification method. When a mission 
failure/mishap occurred during an “Extended Operations” phase, the mission failure/mishap was 
not counted and the mission is only considered to be a “Past (extended) Mission.” Table 3 shows 
that there were 20 mission failures for missions launched from 1994 - 2014 with only 5 of those 
mission failures on missions launched in the past decade (2004-2014). 



Table 3. Overview of Mission information collected for NASA Missions launched from 1994 - 2014 and 2004 - 2014 including Protoflight (PF) and Full Qualification Mission 
information and mission failures. 


Launch Date 

# All 
Mission 

# All Missions 
w/ Risk Class or 
Protoflight 
information 

#Operating 

Missions 

(Not 

Extended) 

((Past 

Missions 

(Not 

Extended) 

(♦Extended 

Past 

Missions 

(♦Extended 

operating 

Missions 

#AII PF 
Missions 
(Operating, 
Past, 

Extended) 

(♦Operating PF 
Missions (Not 
Extended) 

#Past PF 
Missions 
(Not 

Extended) 

(♦Past 

Extended PF 
Mission 

♦(Operating 
Extended PF 
Missions 

#Mission 

Failures 

#PF Mission 
Failures 

Known PF 

Linked 

Failures 

1994 - 2014 

215 

110 

36 

73 

19 

41 

69 

16 

17 

8 

28 

20 

5 

0 

2004-2014 

84 

83 

24 

32 

5 

23 

50 

16 

12 

3 

19 

5 

3 

0 


Table 4. Example of Mission information collected for NASA Missions launched from 1994 - 2014 including Protoflight and Full Qualification subsystem information, risk mission 
classification, launch dates, current phase and responsible government agency and or NASA Center. 


Mission Name 

Mission 

Class 

Protofl ight systems 

Proto-Qual Systems 

Non-Protoflight 

systems 

Unsure Systems 

Launch Date 

Current Phase 

Government 

Agency/Center 

JUNO 

A 





8/5/2011 

Operating 

NASA JPL 

Gravity Recovery and Interior 
Laboratory (GRAIL) 

C 




LGRS 

9/10/2011 

Operating 

NASA J PL 

Suomi NPP 

B 




ATMS 

10/28/2011 

Operating 

NASA GSFC 

MSL 

A 




MMRTG 

11/26/2011 

Operating / 
Extended 

NASA JPL /GSFC 

NuSTAR 

C/D 





6/13/2012 

Operating / 
Extended 

NASA JPL /NASA GSFC 

RBSP (Van Allen Probes) 

C 

EFW 




8/30/2012 

Operating 

Johns Hopkins University 
APL 

TDRS-K 

A 





1/30/2013 

Operating 

NOAA / NASA GSFC 

LDCM Landsat 8 

B 

LDCM 




2/11/2013 

Operating 

NASA GSFC / US geological 
Survey 

IRIS 

C/D 

Spacecraft 




6/26/2013 

Operating 

NASA GSFC /ARC 

LADEE 

C/D 

Spacecraft 




9/6/2013 

Past 

NASA ARC 

MAVEN 

B 



Solar Arrays 


11/18/2013 

Operating 

NASA GSFC /CU-LASP/ SSL 

TDRS-L 

A 





1/23/2014 

Operating 

NOAA / NASA GSFC 

Global Precipitation 
Measurement (GPM) 

B 

Dual-frequency 
Precipitation Radar 
(DPR) 




2/27/2014 

Operating 

JAXA/ NASA GSFC 

OCO-2 

C 



Spacecraft 


7/2/2014 

Operating 

NASA JPL 

Orion EFT-1 


EFT-1 




12/5/2014 

Past 

NASA 


Protoflight Mission Failures between 2004 - 2014 


Over the past decade, NASA has launched a total of 84 missions. All but one mission had enough 
accessible information to detennine the flight qualification method used. 50 out of the 83 missions 
are classified as Protoflight qualified missions according to the assumptions made in the 
Protoflight Criteria discussed previously. In other words, 60% of NASA’s missions in the past 
decade have had at least one Protoflight subsystem and/or instrument on board. This is a significant 
increase in the use of the Protoflight method compared to a study completed in 1975 that 
documented 16 out of the 55 missions (or 29%) over a similar time period were Protoflight. [1] 
Today, 35 of the 50 Protoflight missions are operational (16 Operating Phase and 19 
Operating/Extended Phase) while, 15 Protoflight mission are no longer operational (12 Past Phase 
and 3 Past/Extended Phase), Table 3. 

Only 5 missions over the past decade publicly reported a mission failure. Of the 5 reported, 3 
occurred on Protoflight missions. In other words, 6% of the Protoflight missions (3 out of 50) and 
6% of the Full qualification missions (2 out of 33) over the past decade have experienced a 
publically reported mission failure, Table 3. To better understand if there is a link between the 
Protoflight qualification method and a reported Protoflight mission failure, the 3 Protoflight 
missions (Glory, OCO and DART) that reported a mission failure were reviewed. 

In 2011, the Glory satellite was lost in a launch vehicle failure when the payload fairing of the 
Taurus XL T9 launch vehicle failed to separate during ascent according to the Taurus XL T9 
Mission Glory Mishap Investigation Board (MIB). [2] In 2009, the Orbiting Carbon Observatory 
(OCO) mission was lost in a launch vehicle failure when the payload fairing of the Taurus launch 
vehicle failed to separate during ascent according to the OCO Mishap Investigation Board (MIB). 
[3] In 2005, Demonstration of Autonomous Rendezvous Technology (DART) collided with the 
MUBLCOM satellite it was attempting to rendezvous with using completely autonomous 
maneuvers. The collision was found to be caused by inaccurate navigation system performance by 
the DART MIB. [4] None of the three payload mission failures were linked to the payload’s 
Protoflight qualification method. 

In conclusion, the data indicates that the Protoflight approach, in and of it itself, does not increase 
the mission risk of in-flight failure. This conclusion is in agreement with a previously reported 
study conducted in 1975 on Protoflight Spacecraft. [1] 

Protoflight Mission Failures between 1994 - 2014 


Documentation regarding the risk classification and qualification method used was not readily 
accessible for missions launched between 1994 and 2004, resulting in a data set that is neither 
comprehensive nor sufficient. While a large amount of information regarding NASA missions 
from 1994 - 2004 was inaccessible, the data is included for the reader’s edification only. 



Over the past two decades, NASA has launched a total of 2 1 5 missions. Approximately half of the 
launched missions (110 missions) had enough accessible information to detennine the flight 
qualification method used. 69 out of the 110 missions are classified as Proto flight qualified 
missions according to the assumptions made in the Protoflight Criteria previously discussed. 
Today, 44 Proto flight missions are operational (16 Operating Phase and 28 Operating/Extended 
Phase) while 25 Proto flight mission are no longer operational (17 Past Phase and 8 Past/Extended 
Phase), see Table 3. Of the 17 Past Proto flight missions, only 5 Proto flight missions had a mission 
failure reported and published (Glory, OCO, DART, WIRE, ICESat). (Note: While the ICESat 
instrument laser failure was not classified as a NASA Mission Mishap, the failure is being included 
in the Protoflight mission failures due to the nature of the instrument laser failure and for the 
reader’s edification.) 

Infonnation regarding Glory, OCO and DART can be found in the previous section. In 1999, the 
Wide-Field Infrared Explorer (WIRE) was declared a loss only 4 days after launch. The WIRE 
MIB determined that the root cause of the mission failure was due to a digital logic design error in 
the instrument pyro electronics box that caused the release of the telescope cover earlier than 
planned. The report stated that this error may have been caught during the acceptance/qualification 
testing if the test was allowed. However, the mission did not acquire enough pyros to test this 
function. [5] While none of the four previously mentioned Pro to flight payload mission failures 
were li nk ed to the payload’s Protoflight qualification method, the ICESat GLAS laser failure can 
be debated if the failure is linked to the Protoflight qualification method. 

ICESat was successfully launched in January 2003. On March 29, 2003, operating day 36, the 
Geoscience Laser Altimeter System’s (GLAS) Laser 1 (out of three) unexpectedly stopped 
emitting. The laser failure resulted in the fonnulation of the Independent GLAS Anomaly Review 
Board (IGARB) to better understand the failure and to optimize science return. The laser failure 
resulted in a modified operating plan, with an approximate 30 day operation period (campaign), 
three times per year. This meant that the GLAS duty cycle was reduced from 100% to 27% per 
year. [6] To account for the modified operating plan, the science measurement campaigns were re- 
planned, and resumed in September 2003 using Laser 2. Even with the failure of Laser 1 and the 
reduced duty cycle of GLAS, the mission was successful and was granted a 2.5 year extension in 
2005. After seven years of service, ICESat was decommissioned in August 2010. While the Laser 
1 failure resulted in significant re-planning, the mission requirements were met and the mission 
ultimately was successful. [7] 

The IGARB reported several findings that relate to the Protoflight qualification approach and to 
the use of heritage and COTS parts. Below are excerpts from the IGARB Executive Summary. [8] 

1. “It is likely that the same problem that affected Laser 1 may also exist in Lasers 2 and 3. 
... Laser 1 operated for approximately 74 days of pre-launch operations plus 36 days of 
on-orbit operations. Lasers 2 & 3 pre-launch operations were 44 & 37 days, respectively.” 

2. “The potential problem . . .had not been uncovered during GLAS ground testing. The GLAS 
Instrument used a standard Protoflight qualification regime, which included thermal- 
vacuum and vibration environmental tests. Also, an Engineering Test Unit (ETU) had been 
constructed and tested, and accelerated life tests were successfully performed on similar 



laser diode arrays. The detrimental effect of indium contamination is the major factor that 
had not been anticipated, detected, or accounted for. Laser 3 and the Engineering Test Unit 
(ETU) had experienced catastrophic failures of diode arrays during ground testing that had 
been analyzed and repaired with part replacement. Failure analyses conducted at the time 
had not uncovered the gold indide issues, and concluded that the likelihood of failure 
reoccurrence was extremely rare. Also, Lasers 1 and 3 had experienced small power output 
drops during unit level testing that were believed to be diode bar shunts. ...The GLAS 
design incorporated extra diode arrays to allow for the darkening of individual bars by 
shunts.” 

3. “The GLAS laser diode arrays are complex, high technology and high perfonnance 
assemblies that were procured as Commercial Off the Shelf (COTS) products for 
spaceflight applications. . . . The IGARB also concluded that hardware designated as having 
heritage from previous flight missions requires a rigorous analysis to ensure that it is the 
same as previously used and that it meets the requirements of the new mission. Heritage 
cannot be applied when there is a general lack of knowledge of the fabrication and 
assembly process, which is the general case for COTS products and the GLAS laser diodes 
in particular.” 

4. "It was noted by the Project that a significant factor in the anomaly was that GLAS was 
developed as Class C instrument, with Grade 3 parts (utilize manufacturer’s processes and 
controls).” 

Excerpts 1 and 2 from the IGARB Executive Summary are related to the Protoflight qualification 
approach. In regards to excerpt 1, a report documenting the total time that Laser 1, 2 and 3 were 
operated was not available and thus no conclusions can be drawn from the differences in pre- 
launch laser operation durations and the potential effects. Excerpt 2 states that the GLAS followed 
the standard Protoflight qualification regime including thennal-vacuum and vibration 
environmental testing and the failure mode was not detected during this testing. Failures and 
anomalies did occur on GLAS’s ETU. However, the failure analysis in regards to the GLAS ETU 
pointed to different failure modes than those found by the IGARB on the Protoflight GLAS. 
Excerpt 3 and 4 from the IGARB Executive Summary are related to heritage COTS parts. The 
Protoflight qualification method does not state weather or not heritage and/or COTS parts can be 
used and therefore the decision to use heritage and/or COTS parts is not directly linked to the 
Protoflight method. 

The authors do not attempt to draw any conclusions from this set of incomplete data. This section 
was provided for the reader’s edification and may be used in the future as a starting point for 
further analysis and review. 

Discussion 


The use of the Protoflight method is widely accepted and successful within NASA’s flight projects 
spanning from small Category 3 Class D missions all of the way to Category 1 Class A robotic 
missions. In fact over the past decade only 6% of the Protoflight missions (3 out of 50) and 6% of 



the Full qualification missions (2 out of 33) have experienced a publically reported mission failure. 
The low number of mission failures is a result of the Agency and several Centers creating and 
implementing standards and requirements regarding the testing criteria of Protoflight qualified 
subsystems, instruments, payloads and spacecraft based off of the first General Environmental 
Test Specification for Spacecraft and Components (GSFC S-320-G-1) published in 1969. These 
documents have provided NASA Programs/Projects with accepted Proto flight qualification test 
criteria and design margins to ensure successful missions. 

However none of the documents reviewed attempt to address the implementation of the Proto flight 
method. For an example, none of the documents reviewed address the following areas: 

• Relevant heritage of components, instruments, subsystems, etc. 

• Complexity of the mission 

• Multiple units 

• Documenting the decision process for the above items as well as the use of Proto flight in 
tenns of the technical, cost, schedule and risk requirements 

One document addresses the link between risk classification (acceptable risk/risk tolerance) and 
the Proto flight method, NPR 8705.4 Risk Classification for NASA Payloads. Risk classification 
establishes the acceptable risk associated with a program or project which has a direct impact on 
the implementation of the Proto flight method. “1.3.1 With the acceptable risk classification level 
established, using Appendix C as the guideline, the project can define and apply the appropriate 
design and management controls, systems engineering processes, mission assurance requirements, 
and risk management processes...” [9] The Risk Management approach taken by the 
Program/Project effects the implementation of the Proto flight method throughout the Life Cycle 
of the project. Program/Project Managers often strike a balance between technical risk mitigation 
and maintaining cost and schedule commitments. For an example, depending on the risk 
classification, budget, schedule, and heritage one project may or may not use an Engineering Test 
Unit (referred to as an Engineering Model in NPR 8705.4). The authors draw attention to counter 
an engineering culture that seeks to minimize risk by suggesting that it is always more appropriate 
to manage risk, than minimize it. To quote Charlie Bolden, “Put another way, risk intolerance is 
a guarantee of failure to accomplish anything of significance.” [10] 

Variations in the implementation of the Pro to flight method have caused some to define another 
tenn, ProtoQual. Some have used this tenn interchangeably with Protoflight while others believe 
that there is a difference between the two terms. Given the definition of the term Protoflight and 
the various acceptable Protoflight implementations, any qualification method that does not meet 
Full Qualification (meaning 2 nearly if not identical units are built and tested, the 1 st called the 
prototype to Qualification levels and durations and the 2 nd called the flight unit to Acceptance 
levels and durations) is considered Protoflight . To better understand how Pro to flight is viewed, 
the current uses and issues of Proto flight, and the decision making process to use Proto flight within 
NASA, the authors interviewed key NASA members with experience in robotic and human rated 
programs/projects. 



Key NASA members mostly agreed on all topics that were discussed in the interviews. The first 
was that the Pro to flight Method (at the spacecraft, payload, subsystem, and/or instrument level) is 
accepted throughout NASA for robotic missions (including Risk Class A Category 1 robotic 
missions) with the understanding that some missions may wish to perfonn Full Qualification given 
the risk posture of the Program/Project. In other words, Pro to flight is considered a good 
engineering practice supported by analyses and precedent. To paraphrase one NASA civil servant, 
using Protoflight seems very logical when you blend risk, schedule, cost and technical. For 
Programs/Projects that have high risk and high criticality payloads/subsystems/instruments, Full 
Qualification should be considered. Second was the decision making process on the 
implementation and use of Protoflight should be documented . Typically the decision to go to 
Protoflight is typically not documented well if at all. If the decision to use the Protoflight method 
is documented, the documentation typically does not cover why Proto flight was chosen in regards 
to the overall risk posture, the implementation strategy used such as using an ETU or not, the varies 
decisions regarding the qualification compliance matrix, nor does it cover the current environment 
in which the decision was made. All parties agreed that the documentation of this decision would 
be extremely difficult for one main reason. The implementation of Pro to flight can vary throughout 
the lifecycle due to unforeseen issues such as budget cuts, technical and schedule issues. For an 
example, originally a subsystem may start off as Full Qualification but due to schedule and 
technical issues within the project the subsystem switches to use the qualification unit as the flight 
unit making the subsystem Proto flight. Another example, is that the project may start off as 
Pro to flight but due to increased public visibility the project may switch to Full Qualification. 

The appropriateness of Protoflight and the implementation of Protoflight was discussed. The 
authors found that there is no general rule of thumb of when or how a Program/Project should use 
the Protoflight method. The lack of a “prescriptive” Protoflight approach is accepted and allows 
for each Program/Project to openly discuss/debate the various Pro to flight implementations. Some 
of the debates circle around the following questions: 

• Should a Project use the Pro to flight Method for the 1 st unit in a multiple unit mission? 

• When should a Project use the Proto flight Method without an ETU? 

• Should the Category 1 Class A mission use Protoflight subsystems/instruments if relevant 
heritage exists? 

• What relevant heritage is required to ensure it meets this Program/Project requirements? 

• Should the Project reduce design requirements on a Pro to flight unit to save mass and cost? 

• Should a Protoflight Project use the “Building Block design” for qualification (i.e. not a 
single unit used to qualify the entire system, qualified at various levels and over various 
units)? 

• Should a Program/Project use the Pro to flight Method (payload, subsystem, instrument 
level) on human spaceflight missions that are not multiple units? That are multiple units? 

Many of these questions have been answered by previous missions however no project is the same 
and with the lack of documentation describing the rationale behind the decision, it is difficult to 
understand a previous mission’s applicability. 



The decision making process to use Proto flight, the method of Proto flight implementation and the 
qualification methods selected in the compliance matrix is extremely beneficial for the 
Program/Project. During this process the complexity, heritage, cost, schedule, and risk are all 
discussed with respect to one another resulting in an increased shared knowledge of the 
Program/Project. This process helps to draw out the natural tension between maximizing the 
science and mitigating risks while maintaining schedule and budget. One of the main issues 
expressed with this process is the over estimation of relevant heritage designs, components, 
instruments and subsystems. For an example, if an instrument is thought to have substantial 
relevant heritage the instrument may follow the Protoflight path to minimize cost and schedule. 
However, if the relevant heritage was over stated and did not meet the projects requirements such 
as maximum and minimum operating temperatures, substantial rework may be necessary causing 
a slip in the schedule and an increase in cost. The over estimation of the relevant heritage on a 
project is common and leads to rework and in some cases redesigns of key components ultimately 
causing cost and schedule issues within the project. A Project/Program that chooses to use the 
Proto flight approach with relevant heritage, it is extremely important that the Program/Project 
Manager ensures that the current project’s requirements falls within the heritage design, 
component, instrument and/or subsystem’s requirements and that the documentation of the 
relevant heritage is comprehensive and sufficient. 

Conclusion 


The NASA Protoflight Research Initiative is an internal NASA study conducted within the Office 
of the Chief Engineer to better understand the use of Protoflight within NASA. Extensive literature 
reviews and interviews with key NASA members with experience in both robotic and human 
spaceflight missions has resulted in three main conclusions and two observations. The first 
conclusion is that NASA’s Protoflight method is not considered to be “prescriptive.” The current 
policies and guidance allows each Program/Project to tailor the Protoflight approach to better meet 
their needs, goals and objectives. NASA’s current approach to allow each Program/Project to 
implement the Protoflight method based on the Program/Project’s requirements is largely accepted 
and cherished. Due to NASA’s approach, there are many ways to implement the Pro to flight 
method depending on the relevant heritage, robotic or human spaceflight, multiple units, risk 
posture, budget, schedule, technical advancements, and environment. Second, Risk Management 
plays a key role in implementation of the Protoflight approach. Given that Risk Management is 
finding an acceptable level of risk (risk mitigation vs cost/schedule), any deviations from full 
qualification will be based on the level of acceptable risk with guidance found in NPR 8705.4. 
Finally, over the past decade (2004 - 2014) only 6% of NASA’s Protoflight missions and 6% of 
NASA’s Full qualification missions experienced a publicly disclosed mission failure. In other 
words, the data indicates that the Protoflight approach, in and of it itself, does not increase the 
mission risk of in-flight failure. The first observation is that it would be beneficial to document the 
decision making process on the implementation and use of Protoflight. The second observation is 
that If a Project/Program chooses to use the Protoflight approach with relevant heritage, it is 
extremely important that the Program/Project Manager ensures that the current project’s 




requirements falls within the heritage design, component, instrument and/or subsystem’s 
requirements for both the planned and operational use, and that the documentation of the relevant 
heritage is comprehensive, sufficient and the decision well documented. To further benefit/inform 
this study, a recommendation to perfonn a deep dive into 30 missions with accessible data on their 
testing/verification methodology and decision process to research the differences between 
Pro to flight and Full Qualification missions’ Design Requirements and Verification & Validation 
(V&V) (without any impact or special request directly to the project). 
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